Papers with word segmentation task
Building a Word Segmenter for Sanskrit Overnight (L18-1)
Copied to clipboard
| Challenge: | Sanskrit word segmentation is challenging due to the issue of Sandhi . digitisation efforts have made the manuscripts available in the public domain . |
| Approach: | They propose a deep sequence to sequence model that takes only the sandhied string as input and predicts the unsandhized string. |
| Outcome: | The proposed model improves on the current state of the art by 16.79% . the system can be trained "overnight" and be used for production . |
ThaiLMCut: Unsupervised Pretraining for Thai Word Segmentation (2020.lrec-1)
Copied to clipboard
Suteera Seeha, Ivan Bilan, Liliana Mamani Sanchez, Johannes Huber, Michael Matuschek, Hinrich Schütze
| Challenge: | ThaiLMCut is a semi-supervised word segmentation model for word segmenting in Thai . it uses a bi-directional character language model to leverage useful linguistic knowledge from unlabeled data. |
| Approach: | They propose a semi-supervised approach to Thai word segmentation using a character language model. |
| Outcome: | The proposed approach outperforms state-of-the-art models on the benchmark InterBEST2009. |